Skip to content

fix(crawlers)!: Keep statistics and sessions between run() calls - #2449

Draft
Mantisus wants to merge 3 commits into
apify:v2from
Mantisus:keep-state-across-runs
Draft

Mantisus wants to merge 3 commits into
apify:v2from
Mantisus:keep-state-across-runs

Conversation

@Mantisus

Copy link
Copy Markdown
Collaborator

Description

  • A repeated run() continues the crawl: it adds to the statistics of the earlier runs, error trackers included, and keeps the sessions of the session pool. max_requests_per_crawl counts the requests of the earlier runs too, so a run that starts with the limit already reached stops right away. To count from zero, call crawler.statistics.reset() before the next run().
  • The same change for Crawlee JS is proposed in v4: A repeated run() continues the queue but resets the statistics the crawler built crawlee#4397.

Issues

Testing

  • Unit and integration tests for repeated runs with own and passed statistics and session pools.

✍️ Drafted by Claude Code

@Mantisus Mantisus self-assigned this Nov 15, 2026
@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant